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ABSTRACT 

The purpose of this MBA Project was to investigate and provide a comprehensive 
evaluation of the current Department of the Navy Procurement Metrics that are collected. 
These metrics are collected by the Office of the Deputy Assistant Secretary of the Navy 
for Acquisition Management. This project was conducted at their request and with their 
support. The goal of this project was to determine if the current metrics are the 
appropriate procurement perfonnance measures. Specifically, this MBA Project 
attempted to answer these three questions: (1) Do the metrics align with strategy? (2) 
Can they be measured effectively? and (3) Are they linked to value? The framework 
used to explore these questions is Robert Simon’s Levers of Control model. These 
procurement metrics are part of a Diagnostic Control System and are being evaluated as 
such. Simon’s Nature of Measures model is also used in the analysis and helps detennine 
metric objectiveness, completeness and responsiveness. 
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I. INTRODUCTION 


Data collection and availability has opened up many opportunities to use that data 
in a way that helps us become better managers. There are several issues surrounding the 
use of data and specifically metrics. For instance, is the data accurate? Is that data in a 
form that makes it reasonable to expend the energy retrieving it? Is the expended energy 
(resources) worth the information that is gleaned from the data? 

The purpose of this project is to provide an evaluation of the current Department 
of the Navy Procurement performance measurements (metrics) being collected from the 
Major Naval Systems Commands (SYSCOMs). Specifically, the metrics are being 
collected from the Head of Contracting Activities (HCAs) within those SYSCOMs. This 
evaluation will hopefully contribute to deciding if the current procurement metrics are 
useful, appropriate, and of value. 


A. BACKGROUND 

In fiscal year 2002 the SYSCOMs were the subject of a Booz Allen and Hamilton 
effectiveness and efficiency study. The study resulted in eleven recommendations: nine 
for internal spend and two for industry spend. Of these eleven recommendations two 
applied specifically to the procurement community. Within these two recommendations, 
Booz Allen and Hamilton identified an “enabler” for the procurement community. The 
enabler was “define and institute contracting performance measures and reporting” 
(Procurement Metrics Guide [PMG], vl4.3, p. 4). 

The office of the Assistant Secretary of the Navy for Research, Development and 
Acquisition, ASN (RDA), then tasked the office of the Deputy Assistant Secretary of the 
Navy for Acquisition Management, DASN ACQ, to “provide a ‘dashboard’ or 
spreadsheet for examining the effectiveness and efficiency of the Department of the 
Navy’s Material establishment” (PMG, vl4.3, p. 4). DASN ACQ created a working 
group consisting of HCAs representatives from ten SYSCOMs to assist in the 

development of standard procurement performance measures. 
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The working group decided on five focus areas as part of their Balanced 
Scorecard 1 2 approach to developing the procurement performance measures. The five 
focus areas are Customer, People, Process, Financial, and Value. Each area has a 
strategic theme, objectives, and associated metrics. The dashboard was designed for use 
specifically within the Navy’s Procurement Community, both Military and Civilian, and 
to serve as an executive level summary of the Navy’s procurement performance. 

The metrics put together by the working group measure the procurement activities 
of HCA’s representing the following ten SYSCOMs: 


Marine Corps Installations and Logistics (MARCOR I&L) 
Marine Corps Systems Command (MARCORSYSCOM) 
Military Sealift Command (MSC) 

Naval Air Systems Command (NAVAIR) 

Naval Facilities Engineering Command (NAVFAC) 

Naval Sea Systems Command (NAVSEA) 

Naval Supply Systems Command (NAVSUP/FISC’s/ICP’s) 
Office of Naval Research (ONR) 

Space and Naval Warfare Systems Command (SPAWAR) 
Strategic Systems Project Office (SSP) 


B. PROBLEM IDENTIFICATION 

The HCA working group developed 21 metrics that they stated, when used, would 
give an overall picture of the effectiveness and efficiency of the Navy’s procurement 
function. The Office of the DASN ACQ is currently collecting data for the metrics from 
the SYSCOMs as well as pulling data for the metrics from existing databases. However, 
the office of the DASN ACQ is not receiving all of the HCA responsible metrics from 
each of the ten SYSCOMs. Some simply just do not have the infonnation needed to 
answer DASN’s metrics data call or do not have the necessary resources in place to 
retrieve the data. This creates an issue. The metrics need to be evaluated in order to 
determine if they are useful, appropriate, and of value. However, without a relatively 

1 The Balanced Scorecard is a system developed by Robert S. Kaplan and David P. Norton. The 
scorecard takes an organization’s mission and strategy and translates that into goals and measures, both 
financial and non-financial. The basic scorecard uses four areas for measurement: financial, customer, 
internal business, and innovation and learning. The metrics working group adapted the basic scorecard to 
include five focus areas in order to fit their specific needs. 
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complete set of metrics it is difficult to determine whether or not the effort should be 
made to create and generate a full and accurate metric report from each of the ten 
SYSCOMs. 
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II. LITERATURE REVIEW AND FRAMEWORK USED 


A. MANAGEMENT CONTROL SYSTEMS 

Communication is a critical element of the operations of organizations. 
Information of all kinds is communicated through both formal and infonnal channels. 

One reason managers of organizations are interested in the information communicated is 
because it sends signals describing the health of the organization. Communication 
channels also provide a means for the managers to disseminate information. Senior 
managers can use those channels for communicating a strategic vision, profit goals or 
receiving infonnation about organizational developments or perfonnance. 

All of this information serves as one of the controls in the organization. Controls 
can be put in place through many methods, varying from internal control systems focused 
on inventory tracking to a company’s mission statement. In Robert Simon’s book, Levers 
of Control, he “focuses primarily on the informational aspects of management control 
systems—the levers managers use to transmit and process information within 
organizations” (Simons, 1995, p. 5). He uses the following definition of management 
control systems: management control systems are the formal, information-based routines 
and procedures managers use to maintain or alter patterns in organizational activities 
(Simons, 1995, p. 5). 

The information-based management control system involves information 
transmission not only from the top management level to lower levels, but also 
information flow in the opposite direction. The information flow from the lower levels to 
top management allows for the monitoring of implemented strategies and other efforts 
that were communicated down from top management. Figure 1 illustrates this point. 
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Figure 1. Information Needs of Top Managers (Simons, 1995, p. 6) 


As stated above, organizational control can take many forms. A well performing 
organization will likely have some sort of control throughout all facets of their 
organization. This may sound oppressive and stifling to the employee’s innovation and 
motivation. However, this does not have to be the case. Control does not necessarily 
mean micro-managing. 

Robert Simons’ framework has business strategy as its heart. This is how Simons 
describes the framework: 

Business Strategy—how a firm competes and positions itself vis-a-vis its 
competitors—is at the core of the analysis. The second level introduces 
four key constructs that must be analyzed and understood for the 
successful implementation of strategy: core values, risks to be avoided, 
critical performance variables, and strategic uncertainties. Each construct 
is controlled by a different system, or lever, the use of which has different 
implications. These levers are: (Simons, 1995, p. 6) 


1 . belief systems, used to inspire and direct the search for new opportunities; 

2. boundary systems, used to set limits on opportunity-seeking behavior; 
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3. diagnostic control systems, used to motivate, monitor, and reward 
achievement of specified goals; and 

4. interactive control systems, used to stimulate organizational learning and the 
emergence of new ideas and strategies (Simons, 1995, p. 7) 

The following figure is the representation of the Levers of Control, with business 
strategy at the core, surrounded by key constructs to be analyzed and understood, and 
controlled by the four systems, or levers. 


BELIEF 

SYSTEMS 


BOUNDARY 

SYSTEMS 



INTERACTIVE DIAGNOSTIC 

CONTROL CONTROL 

SYSTEMS SYSTEMS 


Figure 2. Levers of Control Framework (Simons, 1995, p. 7) 


B. BELIEF AND BOUNDARY SYSTEMS 
1. Belief Systems 

Belief systems, as defined by Simons, “are the explicit set of organizational 

definitions that senior managers communicate formally and reinforce systematically to 
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provide basic values, purpose, and direction for the organization” (Simons, 2000, p. 276). 
The belief system provides the means of control over the organization’s foundational 
principles and ideals. 

These principles and ideals are often articulated by an organization’s founders and 
provide the basis for employee action and motivation when rules and regulations alone 
will not sufficiently address the issue or situation that employees face. They address such 
things as responsibilities to customers and other employees. This set of principles is 
typically called a set of core values. Simons defines core values as “the beliefs that 
define basic principles, purpose, and direction” (Simons, 2000, p. 276). The core values 
often take the form of a mission statement or credo. 

2. Boundary Systems 

Belief systems go a long way to inspire and motivate employees to achieve. They 
give them a sense of purpose to their work. This is beneficial, but there is a limit to what 
employees can and should do. The employees need to be motivated, innovative and serve 
the customer, but they should not pursue an avenue which opens up the business to undue 
risk. An organization needs to understand what risks they want to avoid and apply a set 
of boundaries to employees to minimize the likelihood of that risk occurring. This lever 
of control, boundary systems, plays an integral role in business strategy. Without 
boundaries the belief system is seriously degraded. Enthusiasm can lead to undesired 
outcomes if left unchecked. The boundary system is like the brakes on enthusiasm and 
innovation. Simons uses the example of brakes on a car: 

Cars have brakes for two reasons. First, and most obvious, they allow the 
driver to slow the car down and stop safely. However, cars have brakes 
for another reason. They give the driver the confidence to go very fast. 

Imagine a high-perfonnance racing car on a speedway. The driver can 
operate at top speeds only if he knows that he can rely on excellent brakes 
to control the car on right turns. Like the fastest cars, managers of high- 
performance businesses need the best brakes to control strategic risks that 
are an inevitable consequence of driving their businesses to their 
maximum potentials (Simons, 2000, p. 275). 

A boundary system basically tells employees what not to do. It lets the 
employees know which actions management deems unacceptable. This may seem like a 
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hindrance on their innovation, but in fact, this is what allows them to be innovative. It 
allows them to be innovative in a way which reduces the likelihood that an organization 
will be exposed to risk. Another approach that a manager could employ would be one 
telling employees what to do. This method ensures that employees do not perform 
actions that top management feels are risky. This system also limits what individuals can 
actually do. Now that the employees are told what to do, they will come to work, follow 
procedures, and leave. There is minimal opportunity for innovation within this system. 

C. INTERACTIVE AND DIAGNOSTIC CONTROL SYSTEMS 

The following two systems, which complete the levers of control model, mean 
quite literally what their titles portray. Managers use these two systems in quite different 
ways. Interactive control systems require a good deal of interaction between top 
managers and employees discussing possible emerging opportunities or threats to their 
business given the changing market environment. The system’s intent is to use the 
information gained from these interactions to modify strategy as the environment 
changes. Diagnostic control systems are used to diagnose issues or discrepancies which 
vary from the intended strategy or expected output of the organization. Diagnostic 
control systems are used to monitor performance of critical perfonnance variables and 
monitor the implementation of the organization’s intended strategy. 

1. Interactive Control Systems 

Interactive control systems “are the formal infonnation systems that managers use 
to personally involve themselves in the decision activities of subordinates. Simply stated, 
[they] are the hot buttons for senior managers. They provide the infonnation that the 
boss pays a lot of attention to and are used to create an ongoing dialogue with 
subordinates” (Simons, 2000, p. 216). 

Interactive control systems are centered on “strategic uncertainties” or the 

“emerging threats and opportunities that could invalidate the assumptions upon which the 

current business strategy is based” (Simons, 2000, p. 215). These uncertainties, by 

definition, are not known beforehand, they emerge over time. The environment is 

constantly evolving and changing and then so too does the organization if it is to survive. 

Therefore, business strategy, the core of the levers of control model, has to be reviewed 
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and examined to determine if the organization’s current strategy will put it in a position 
to be successful. Top managers need to interact with their employees to uncover these 
strategic uncertainties. 

Examples of strategic uncertainties are rapidly changing technologies which could 
possibly render the organizations’ value proposition worthless and governmental relaxing 
or tightening of regulations. In the first case, this represents a threat to the organization, 
but strategic uncertainties, as stated above, can be avenues for opportunities for the 
organization as well. Both could be true for the second situation, changing government 
regulations. This could help or hurt the organization. 

One way to identify these uncertainties is to create an environment of open 
dialogue throughout the organization which focuses on examining and revising the 
business strategy given the complex and changing environment. Everyone needs to be on 
board and focus on the questions that need to be asked, thereby ensuring a successful 
future. This open-dialogue environment is created through an interactive control system. 

The use of interactive control systems involves thorough evaluation of internal 
reports and efforts and questioning subordinate’s assumptions in those reports in an effort 
to challenge old ways of thinking. This challenge should spark debate and dialogue and 
ultimately lead to a pooling of information, gained throughout the organization, for top 
managers to pour over and help guide the identification of strategic uncertainties. This 
method of interactive information gathering and use of that information is very dissimilar 
to how a diagnostic control system is used. 

2. Diagnostic Control Systems 

This research project focuses on the final lever of control, the diagnostic control 
system. Simons defines a diagnostic control system as the “formal information systems 
that managers use to monitor organizational outcomes and correct deviations from preset 
standards of performance” (Simons, 2000, p. 209). The purpose of a diagnostic control 
system is to enable management by exception. The care and attention used to construct 
the system and the validity of the system determines whether or not managing by 
exception can be successful. This system is only as good as the measurements used to 
build it, as will be seen later. 
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The basic premise of a diagnostic control system is that of variance analysis. This 
system compares actual output or results with those that were expected or set as goals. If 
the results are within the bounds of what was intended to be accomplished then the 
manager need not search any further because things are on track. If, on the other hand, a 
result is off target, the manager will note the variance and subsequently investigate 
possible reasons why the deviations occurred. The process is referred to as feedback. 

This is equivalent to a “cybernetic feedback model,” shown in Figure 3. In such a model, 
every measure has a benchmark or standard that represents the expected output. The 
deviation that is the result of comparing the output to the standard provides feedback to 
the input or process parts of the model. The purpose of the feedback is to ensure that 
performance targets are met in the future by adjusting the inputs. 


Standard 


Inputs 



Process 

c 

i 

l 

K 


i 



Outputs 


Feedback 


Figure 3. Cybernetic Feedback Model (Simons, 2000, p. 61) 

A diagnostic control system is applicable in many situations. These situations 
include using tools such as balanced scorecards, budgets, profit plans and standard cost¬ 
accounting systems in a diagnostic manner. Simons argues that there are two key reasons 
for a using diagnostic control system: “to implement strategy effectively and conserve 
scarce management attention” (Simons, 2000, p. 209). 

Time is scarce for managers. There are thousands of tasks and limited hours in 
the day. The goal of a diagnostic control system is to encapsulate relevant variables into 
a user friendly fonnat that managers can use in a time-efficient manner to give them a 
snapshot of the health of their organization. Simons uses the example of the speedometer 
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in your car. During a long trip it would be incredibly tiresome to be constantly 
monitoring the speedometer and making small adjustments to your speed. All of the time 
and energy would be used doing this task. Instead, the car can be put on cruise control 
which frees the time to focus on other things. This is in essence what a diagnostic control 
system will do. The parameters are set and only when the output falls out of range is 
there a signal. When that occurs, then focus should be directed to get the output back 
within bounds. 

The particular variables chosen to be measured must be carefully selected. 

Simons calls these variables, critical performance variables. He defines them as “those 
factors that must be achieved or implemented successfully for the intended strategy of the 
business to succeed” (Simons, 2000, p. 209). The diagnostic control system is in place to 
monitor whether the strategy, through these critical performance variables, is being 
implemented. This system li nk s the strategy to the measurable perfonnance variables. 
The diagnostic control system is crucial to communication and effectively implementing 
strategy in large organizations. 

3. Risks in Diagnostic Control Systems 

The hands off nature of a diagnostic control system, after extensive attention and 
work to ensure a properly constructed and aligned system, is a very desirable attribute. 
However, this system is not without its risks. Simons identifies three risks associated 
with using a diagnostic control system; “measuring the wrong variables, building slack 
into targets, and gaming the system” (Simons, 2000, pp. 212-213). 

Measuring the Wrong Variables People pay attention to things upon which they 
are evaluated. If senior management identifies incorrect or misaligned variables to be 
measured, then subordinates will be concentrating on achieving goals that do not 
significantly contribute to the overall intended business strategy. Measuring the wrong 
variables could hinder or derail strategy implementation. 

Building Slack into Targets If employees’ performance is evaluated on the 
achievement of a goal, then their natural tendency will be to increase the likelihood that 
they achieve that goal. This can be achieved in one of two ways: (1) through hard work 
or (2) by setting easily attainable goals. Managers need to be cognizant of the latter 

behavior and ensure that the goals that are set are challenging ones. 
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Gaming the System In general, employees will work hard towards 
accomplishing goals upon which they are measured. This hard work can be focused 
away from achieving the desired goal in the manner envisioned by top management. 

“This misdirected effort is called gaming” (Simons, 2000, p. 213). Simons (2000) 
identifies three common gaming activities: smoothing, biasing, and illegal acts. The act 
of “smoothing” attempts to show better performance by manipulating the timing and 
recording of transactions. An example of this might occur if the goals for small business 
contract awards (in a given fiscal year) are reached. Rather than make the current year 
look better by recording additional small business contract awards, after the goal is met, 
the employee may chose to record any additional contract awards towards next year’s 
goals. “Biasing” occurs when managers chose to report goals that have been achieved, 
but downplay or bury bad news or goals that were not achieved. The attempt is to control 
and bias the information flow. Finally, pressure to achieve can sometimes prove to be 
too much. Individuals may commit “illegal acts” in order to meet their goals and earn a 
bonus. These acts might violate the law or company policy and therefore would be 
considered illegal. 

D. SELECTING PERFORMANCE MEASURES/METRICS 

In order to mitigate these risks, it is imperative that the correct perfonnance 
measures or metrics are chosen. Performance measures may seem to be intuitive and 
rudimentary in nature, but the opposite is true in many situations. The performance 
measures selected have profound meaning not only for the managers who will be using 
them to make decisions, but more importantly, for those who are being measured. 

Robert Simons identifies three tests which a perfonnance measure must be 
subjected to in order to detennine if the measure is a suitable one: (1) Does the measure 
align with strategy? (2) Can it be measured effectively? (3) Is it linked to value? (Simons, 
2000, p. 234) 

Test one is whether a measure is aligned with strategy. When a metric or measure 
is created, the managers creating it must know that by doing so they are telling their 
employees what is important. Why have the measure if it is not meaningful? Thus, a 
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measure must be in congruence with the strategy of the organization. This not only 
reinforces the organization’s strategy with employees, but it inherently supports the 
achievement of the strategy. 

Just as important as its alignment with strategy, a metric must be able to be 
measured effectively, Test 2. There is little or no value to having a metric that is 
supposed to be telling managers one thing, but is giving them something different. In 
reality the metric might either be too difficult or complex to measure or simply the wrong 
metric. In the later case, the correct metric can replace the incorrect one, but the former 
case is a more difficult one. It requires a more in-depth analysis of the strategy and a 
determination if the ability to capture the essence of the measure exists. 

In Robert Simons’ discussion of Test 2, can it be measured effectively, he 
mentions three distinct adjectives against which measures can be evaluated. He states, 
“Ideally, measures should be objective, complete, and responsive.” (Simons, 2000, p. 

235) These are referred to as the “Nature of Measures.” 

Managers must have solid information on which to base their decisions. Solid 
information comes from having a clear formula for the metric and unambiguous inputs to 
the formula. When the metric is objective it can be independently verified. 2 If the metric 
is a subjective one, then Simons goes on to say in that situation “trust must be high, 
because the subordinate must have confidence that the subjective judgment is fair and 
will be used appropriately.” (Simons, 2000, p. 236) 

Completeness and responsiveness are not as clearly ascertained. Metrics can have 
varying degrees of each. A metric is an attempt to explain an aspect of the organization. 
How well the metric explains that particular aspect of the organization is its degree of 
completeness. When a metric is complete it captures all of the relevant attributes of 
success. A responsive metric is one that responds to the actions of managers. The 
process of feedback and variance analysis used in a diagnostic control system relies 
partly on an organization’s ability to make corrections to inputs to decrease the variance 
between the output and the perfonnance standard. If the metric is not responsive then 

2 An objective metric is not the only type of measure that is acceptable. Although a subjective metric 
may not be independently verified to the same extent as an objective one, a subjective metric can be very 
valuable in some cases. 
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managers can not influence the output. This may have negative impacts on morale 
because it can create a situation where managers are being evaluated on their ability to 
meet perfonnance targets, but do not have the ability to directly influence the results. 

A classic example which illustrates responsiveness and completeness is the “sales 
calls” example. A manager chooses to evaluate employee sales performance by 
measuring the number of sales calls made each week. The metric is responsive because 
the employee has influence on it. While this can be good, it can also create risk. So, if 
employees know they are being evaluated on the number of sales calls per week then they 
can easily increase the number of sales calls without regard for making an actual sale at 
each stop. Because this metric fails to capture the key factor of actual sales it fails the 
completeness test. 

Typically, lower-level jobs which require more routine work can achieve metrics 
that are objective, complete and responsive in nature. The higher up in the managerial 
chain, the more difficult it is to balance all three. The Nature of Measures figure used by 
Simons (Figure 4) provides a template to follow when evaluating objectivity, 
completeness, and responsiveness of perfonnance metrics. 
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Figure 4. Nature of Measures (Simons, 2000, p. 236) 


Lastly, the metric has to be linked to value. The information contained in the 
metric should be useful for the managers and should aid in decision making. The 
information provided by the metrics should enable managers to be more effective and in 
the end, run a more effective organization. In essence, the metrics should provide 
valuable information useful in running the business and infonnative in getting from “A to 
Z.” 

In order to maximize the benefits gained from measuring performance, managers 
must take care in selecting and evaluating specific perfonnance metrics. Specifically, 
performance metrics should be aligned with strategy, be objective, complete and 
responsive, and be linked to value. 
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III. METHODOLOGY 


The author conducted a series of interviews with the Department of the Navy 
Procurement Metrics points of contact provided by DASN ACQ. DASN ACQ provided 
points of contact for the ten SYSCOMs reporting on the metrics. Eventually six-face-to- 
face interviews were performed over the course of three days. The interviews varied 
from one-on-one, one-on-two, and occasionally involved talking to three or more people. 
The additional individuals were identified by the points of contacts as having valuable 
insights about the metrics. In total, the author interviewed thirteen individuals face-to- 
face. The interviews were conducted at six SYSCOM sites and lasted, on average, an 
hour and a half. There were no telephone interviews or follow-up interviews. 

The interviewees had varied familiarity with the performance metrics and job 
experience. This ranged from approximately 2-3 weeks as the DoN Procurement Metrics 
point of contact to being involved in the process throughout the history of the metrics 
effort. Regardless of this factor, every interviewee offered insight into the metrics. Each 
interviewee had his or her own specific tone about the metrics; however commonality of 
issues with the metrics was apparent across all of the interviews. 

The general framework used to evaluate the metrics consists of three interview 
questions: (1) Do the metrics align with strategy? (2) Can the metrics be measured 
effectively? and (3) Are the metrics li nk ed to value? The interviews provided key 
information that was useful in answering all three of these questions. The information 
also provided general views on how the overall metrics process could be improved and 
specifically how the metrics could become more valuable for the HCAs. 

Table 1 is a summary of the DoN Procurement Metrics evaluated in this project. 
The following section summarizes the interviewees’ and author’s evaluation of the 
current performance metrics. The author’s evaluation of the metrics represents the 
synergy of perfonnance measurement theory and information gained during the 
interviews. 
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FOCUS 

AREA 

TIER 1 METRIC 

TIER 2 METRIC 

TIER 3 
METRIC 

TIER 4 
METRIC 

DESCRIPTION / FORMULA 

FREQUENCY 

COLLECTED 

BY 

Customer 

Customer Satisfaction Index 




Average Index from Customer Satisfaction Survey 

Annual 

HCA 

People 

Human Capital Index 

Employee Survey 



Average Index from Employee Satifaction Survey 

Annual 

HCA 

Workforce Stability 



Percent difference between authorized end-strength 
and actual employees onboard 

Quarterly 

HCA 

Qualifications 

Continous 

Learning 


Percent employees meet their CL requirements and 
hold valid certificates 

Annual 

DASN ACQ 

DAWIA 


Percent DAWIA certified to or above level required by 
their position 

Quarterly 

DASN ACQ 

Acquisition 

Professional 

Community 


Percent of incumbents of CAPs who are APC 
members 

Quarterly 

DASN ACQ 

Process 

Process Improvement Index 

E2E Metrics 



Percent of automated procurement transactions 
(defined on PEO-ARBS website) 

Quarterly 

DASN ACQ 

Efficiency 

Improvement 

P-Card 

Delinquency 


60 days delinquent/total transactions 

Quarterly 

DASN ACQ 

DD1716 


Average turnaround time from receipt to disposition of 
DD1716 (notice of contract deficiencies) 

Quarterly 

DASN ACQ 

Interest 

Penalties 


Ratio YTD interest penalties/penalties same period 
previous FY 

Quarterly 

DASN ACQ 

Cycle Time 



Average cycle time as the term is defined in glossary 
for ACAT 1 and II Program contracts 

Annual 

HCA 

Consolidate Services 
Contract 



Services contracts awarded YTD/awarded previous 

FY 

Quarterly 

DASN ACQ 

Financial 

Contribution Index 

Procurement 
Direct/Indirect Ratio 



Procurement direct labor dollars/total procurement 
labor dollars 

Annual 

HCA 

Industry Spend 
Analysis 

Competition 


Percent dollars awarded competitively/total available 

Quarterly 

DASN ACQ 

Small Business 


Percent dollars awarded to small businesses/total 
dollars awarded and available for small business 

Quarterly 

DASN ACQ 

Commercial 

Items 

Actions 

Percent contracts awarded for commercial items/total 
contracts awarded 

Quarterly 

DASN ACQ 

Dollars 

Percent dollars awarded for commercial items/ dollars 
awarded for commercial items in '99 

Quarterly 

DASN ACQ 

Performance 
Based Services 
Contracting 

Actions 

Percent actions awarded for performance based 
services/total actions awarded for services 

Quarterly 

DASN ACQ 

Dollars 

Percent dollars awarded for performance based 
services/total dollars awarded for services 

Quarterly 

DASN ACQ 

Value 

Product Unit Costing (PUC) 

Large Contracts 



Procurement salary dollars/contract action 
dollars*(customer service survey index) 2 , for contracts 
> $100k 

Annual 

HCA 

Simplified Acquisition 
Procedures 



Procurement salary dollars/contract action 
dollars*(customer service survey index) 2 , for contracts 
< $100k 

Annual 

HCA 


Table 1. DoN Procurement Metrics Summary 
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IV. PROCUREMENT METRICS ANALYSIS 


The Procurement Metrics analysis is divided into four sections. The first three 
sections answer the three questions described as the general framework for evaluating the 
metrics: 1) Do the metrics align with strategy? 2) Can the metrics be measured 
effectively? and 3) Are the metrics linked to value? The fourth section of this chapter 
describes other findings that do not specifically address the three questions. 


A. DO THE METRICS ALIGN WITH STRATEGY? 


The metrics are categorized into five focus areas; Customer, People, Process, 
Financial, and Value. Each focus area has an associated strategic theme. The first 
question evaluates whether the metrics link back to the overall strategy of each focus 
area. If they are not aligned with strategy then there is a fundamental disconnect. 

1. Customer 


Metric 


Customer Satisfaction Index 


The Customer focus area uses only one metric, the Customer Satisfaction Index. 
The strategic theme is to “Deliver unprecedented offensive power, defensive assurance 
and operational independence to the Joint Force Commanders” (PMG, vl4.3, p. 7). Put 
another way, the strategy is to “give the customers what they want.” The index is an 
average of the scores from customer satisfaction surveys which address four specific 
customer satisfaction themes identified in the Procurement Metrics Guide. The Customer 
Satisfaction Index metric is aligned with strategy; the SYSCOMs want to know how well 
they are serving their customers. However, this does not mean that the metric is flawless. 
The metric is still subject to the tests of whether it is li nk ed to value and can be accurately 
measured. 
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2. People 


Metrics 

Employee Survey 


Workforce Stability 


Continuous Learning 


DAWIA Certification 


Acquisition Professional Community 


The People focus area has five associated metrics. The five metrics are the 
Employee Survey, Workforce Stability, Continuous Learning, DAWIA Certification, and 
Acquisition Professional Community. The strategic theme is to “Maintain world-class 
workforce” (PMG, vl4.3, p. 7). All of these metrics align with strategy. The employee 
survey is intended to measure the employee’s overall satisfaction with the organization 
which is an important component of maintaining a world-class workforce. A satisfied 
workforce is likely to be more stable and keeping the workforce stable (providing they 
are well trained) is certainly a part of maintaining a world-class workforce. The last three 
metrics are in place to ensure that employees have the knowledge and necessary skills to 
be effective in their positions, making Workforce Stability that much more meaningful. 
Continuous Learning and the Defense Acquisition Workforce Improvement Act (DAWIA) 
metrics measure the percentage of employees who have their required certification. 
Continuous Learning certification requires job relevant “learning” to be achieved each 
year. The level of DAWIA certification needed depends on the position that the employee 
holds. In order to be certified, employees need to be certified to their level position or 
higher. The Acquisition Professional Community metric, collected quarterly, identifies 
the percentage of incumbents in Critical Acquisition Positions who are members of the 
Acquisition Professional Community (APC) compared to the total number of incumbents 
in Critical Acquisition Positions. The PMG states, “Selection to the APC is based on 
meeting specific training, education, experience and grade requirements.” These 
requirements are controlled by the Director of Acquisition Career Management (DACM). 
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3. Process 


Metrics 

E2E Metric 


P-Card Delinquency 


DD1716 


Interest Penalties 


Cycle Time 


Consolidate Service Contract 


The third focus area is Process. There are six associated metrics in this focus 
area. The strategic theme for Process is to “Continuously improve efficiency” (PMG, 
vl4.3, p. 7). The metrics associated with the Process focus area are E2E Metrics, P-Card 
Delinquency, DD1716, Interest Penalties, Cycle Time, and Consolidate Service Contract. 

Business transactions are moving in the direction of paperless transactions. The 
Navy has set up a website (www.peoarbs.navy.mil) which it refers to as the Paperless 
Acquisition Office of the Navy. The goal of the website is “to simplify and moderni z e 
the Navy acquisition process in the area of contract writing, administration, finance and 
auditing.” The E2E Metric measures the percentage of total procurement transactions 
that are automated, or the ratio of automated procurement transactions over total number 
of procurement transactions. Automation in this case improves efficiency because it 
reduces time spent filling out and sending paper procurement transactions. This metric 
li nk s to the strategy of the Process focus area. 

The DoD Purchase Card Program Management Office sent out a memorandum on 
17 SEP 1999 regarding the issue of Purchase Card Delinquency policies and the need to 
keep delinquent payments (over 60 days) to a minimum. The P-Card Delinquency metric 
is the percentage of dollars of delinquent P-card payments compared to the total purchase 
card dollars. The metric is linked with the overall strategy for the Process focus area 
because decreasing delinquent P-Card payments results in an increased efficiency of 
payments. This metric is consistent with DoD policy as well. 

The DD1716 Metric measures the average turnaround time of the DD1716 form, 
which is a contract deficiency report. The quick resolution of the deficiency can limit the 
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amount of interest charged. The sooner the turnaround time on these forms the less 
money that has to be paid out and more efficient the process which directly links to the 
strategy. 

Interest penalties cost the Navy and the DoD as a whole millions of dollar every 
year. Paying interest is essentially wasted money that results from inefficient practices 
and/or contract mistakes. Not only does the Interest Penalty metric provide the 
opportunity to monitor dollar savings, but it is also a means to monitor if the process is 
becoming more efficient. Increasing dollar savings and efficiency work hand in hand 
towards the ultimate objective: an efficient environment which becomes an effective cost 
savings tool. This metric is collected quarterly and compares the current year’s interest 
penalties paid to date to the same period of the former fiscal year. The output is a ratio 
which hopefully falls below 100 percent. This metric, as well as Consolidate Service 
Contract and the Industry Spend Analysis—Commercial Items (actions & dollars) metrics 
compare current year to last year, so the base needs to be adjusted every year. 

Cycle time is very important in the procurement world and there are many factors 
that influence it (e.g., technological maturity of the proposed system). Reducing cycle 
time can save money. More importantly, a reduction in cycle time gets the products or 
services to the war fighter sooner. A reduction in cycle time can give the United States 
military a distinct advantage over other militaries. The Cycle Time metric is defined as 
“total time in days between the acquisition start date and end date (only applicable to 
AC AT I and II acquisitions)” (PMG, vl4.3, p. 17). Definitions of the start and end date 
are available in the Procurement Metrics Guide. The Cycle Time metric is linked to the 
Process focus area strategy. 

The Consolidate Service Contract metric is the last one under the Process focus 
area. This metric is collected quarterly and is measured in a manner that is similar to the 
Interest Penalty metric. It is a ratio of the number of services contracts awarded year-to- 
date as compared to the same time the previous year. The goal for the ratio is for it to be 
less than 100 percent, indicating that some of the services contracts have been 
consolidated or eliminated. This metric is aligned with the overall strategy of the Process 
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focus area because fewer contracts will have to be maintained and administered and this 
increases the potential for more efficient operations. 


4. Financial 


Metrics 

Procurement Direct/Indirect Ratio 


Industry Spend Analysis—Competition 


Industry Spend Analysis—Small Business 


Industry Spend Analysis—Commercial Items—Actions 


Industry Spend Analysis—Commercial Items—Dollars 


Industry Spend Analysis—Performance Based Services—Actions 


Industry Spend Analysis—Perfonnance Based Services—Dollars 


The Financial strategic theme is to “Maximize return on taxpayer investment” 
(PMG, vl4.3, p. 7). The financial focus area has seven associated metrics. The metrics 
are (1) Procurement Direct/Indirect Ratio, (2) Industry Spend Analysis — Competition, (3) 
Industry Spend Analysis—Small Business, (4) Industry Spend Analysis—Commercial 
Items-Actions, (5) Industry Spend Analysis—Commercial Items-DoIIars, (6) Industry 
Spend Analysis—Performance Based Sendees Acquisition Contracting-Actions, and (7) 
Industry Spend Analysis—Performance Based Services Acquisition Contracting-Dollars). 

The Procurement Direct/Indirect Ratio metric is an attempt at measuring the 
“bang for the buck” of each SYSCOM when it comes to how much support is needed to 
perform the procurement function. The metric is a ratio of the total number of direct 
labor employees as compared to the total number of employees in the procurement 
organization. Direct labor employees are defined as “employees whose primary 
responsibilities are directly involved with efforts required to perform the procurement 
function” (PMG, vl4.3, p. 17). The idea is to provide the necessary support to efficiently 
and effectively accomplish all of the necessary contracting activities while keeping the 
number of supporting positions under control. The strategy of maximizing the return on 
taxpayer investment follows logically, but issues with this metric are discussed in later 
sections. 

The Industry Spend Analysis—Competition metric has a direct link to the strategy 
of the Financial focus area, maximize return on taxpayer investment. This metric is the 
ratio of total procurement dollars awarded competitively as compared to the total 
procurement dollars available for competition. The benefits of competition; cost 
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reduction, improved quality, and innovation, are widely known. There are instances of 
course when competition is not the most advisable way to proceed (e.g., when only one 
source is able to provide the product or service, when time is a critical factor, or when a 
dependable long-term relationship is necessary). 

The Industry Spend Analysis—Small Business metric links to the strategy of the 
Federal Acquisition Regulations regarding small business opportunities for government 
contracts. This metric does not clearly link to the financial focus area strategy of 
maximizing taxpayer investment. However, it does reflect social responsibility and the 
entrepreneurial spirit captured in the Small Business Act. The spirit of the Small 
Business Act is to encourage small business participation in government contracts and 
specifically, “small business concerns shall be afforded an equitable opportunity to 
compete for all contracts that they can perform to the extent consistent with the 
Government’s interest” (Federal Acquisition Regulation [FAR], 19.202-1). 

The Federal Acquisition Regulation also states: 

It is the policy of the Government to provide maximum practicable 
opportunities in its acquisitions to small business, veteran-owned small 
business, service-disabled veteran-owned small business, HUBZone small 
business, small disadvantaged business, and women-owned small business 
concerns. The Small Business Administration (SBA) counsels and assists 
small business concerns and assists contracting personnel to ensure that a 
fair proportion of contracts for supplies and services is placed with small 
business (FAR, 19.201). 

The Industry Spend Analysis—Commercial Items metrics (Actions & Dollars) 
both are linked to the Financial focus area strategy as well as that of the Federal 
Acquisition Regulations. Some of the benefits of using Commercial and Non- 
Developmental items (CANDI) is the speed at which these items can be procured, cost 
savings, and proven technology which mitigates risk. Using CANDI for technologically 
driven products maximizes taxpayer investment because often times commercial demand 
for these items may quicken the evolution and fielding beyond the pace at which 
government research and development could produce the same items. The Actions metric 
is the “ratio of the number of contracts awarded year to date for commercial items as 


24 



compared to the total number of contracts awarded” (PMG vl4.3, p. 13). The Dollars 
metric is the “ratio of HCA total product dollars awarded year to date for commercial 
items as compared to the total product dollars awarded in 1999, during the comparable 
period, for commercial items” (PMG, vl4.3, p. 13). 

The Federal Acquisition Regulation states: 

Agencies shall—(a) Conduct market research to determine whether 
commercial items or nondevelopmental items are available that could 
meet the agency’s requirements; (b) Acquire commercial items or 
nondevelopmental items when they are available to meet the needs of the 
agency; and (c) Require prime contractors and subcontractors at all tiers to 
incorporate, to the maximum extent practicable, commercial items or 
nondevelopmental items as components of items supplied to the agency 
(FAR, 12.101). 

The Industry Spend Analysis—Performance Based Services Acquisition (PBSA) 
Contracting metrics (Actions and Dollars) are aligned with focus area strategy as well as 
DoD policy. The motivation for these metrics is that people working within contracting 
do not need to get bogged down in the details of writing specific processes of how a 
service should be perfonned. The contractor is supposed to be the expert, so give the 
contractor specifics on what the end result of the service should be, but not the minutia of 
how to do it. The effects of PBSA should be cost savings and a more effective and 
efficient use of limited DoD resources because the focus shifts to execution. 

PBSA involves acquisition strategies, methods, and techniques that 
describe and communicate measurable outcomes rather than direct 
performance processes. It is structured around defining a service 
requirement in terms of performance objectives and providing contractors 
the latitude to determine how to meet those objectives. Simply put, it is a 
method for acquiring what is required and placing the responsibility for 
how it is accomplished on the contractor (Guidebook for PBSA in the 
Department of Defense, p. 6). 

The PBSA Actions metric is defined as the ratio of HCA total PBSA awarded 
actions as compared to total awarded actions for services. The PBSA Dollars metric is 
defined as the ratio of HCA total dollars awarded for PBSA contracts as compared to 
HCA total dollars awarded for all services contracts. 
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5. 


Value 


Metrics 

Performance Unit Costing—Large Contracts 


Performance Unit Costing—Simplified Acquisition Procedures 


The fifth and final focus area is Value. The strategic theme of the Value focus 
area is the same as the Financial focus area: Maximize return on taxpayer investment. 

The objective for this focus area is to “Balance quality, cost and productivity” (PMG, 
vl4.3, p. 8). There are two metrics in this focus area and both support the strategic 
theme. The metrics are Performance Unit Costing (PUC)—Large Contracts and PUC — 
Simplified Acquisition Procedures. Both metrics use the same formula: HCA total 
procurement salary dollars (C), divided by the product of contract action dollars (O) and 
the square of the customer satisfaction index (I ). The resultant formula is: [C/(0*I )]. 
The two metrics have different definitions for “O,” the contract action dollars, because of 
the threshold between Large Contracts and Simplified Acquisition Procedures. The 
Simplified Acquisition Threshold is defined by the FAR as follows: 

Simplified acquisition threshold means $100,000, except for acquisitions 
of supplies or services that, as determined by the head of the agency, are to 
be used to support a contingency operation or to facilitate defense against 
or recovery from nuclear, biological, chemical, or radiological attack 
(FAR, 2.101). 

The PUC-Large Contracts metric is measured by taking “HCA total procurement 
salary dollars (C) divided by contract action dollars (O) for contracts valued at greater 
than $100k multiplied by the square of the customer satisfaction index (I): [C/(0*I 2 )]” 
(PMG vl4.3, 14). The PUC-Simplified Acquisition Procedures metric is calculated the 
same way, just using the “$ 100k and less” threshold to determine the “O” variable. The 
motivation for these metrics is to mesh the “bang for the buck” (cost and productivity) 
with the perceived quality by the customers. The output is a number (e.g. .004959716), 
which by itself, is of no use. It can be of some use if several other factors are taken into 
account such as the validity of the Customer Satisfaction Index, the ability to accurately 
collect total procurement salary dollars and having several data points (a trend) to 
analyze. 
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B. 


CAN THE METRICS BE MEASURED EFFECTIVELY? 


All of the DoN metrics appear to be aligned with the strategies of the focus areas. 
One of the strategies is to give the war fighters what they need when they need it and a 
survey could possibly capture all that is needed to ascertain that, but it could also fall 
short. In this section, the metrics will be evaluated on their objectivity and completeness 
as described in the literature review. A measure is objective when it can be 
independently verified and is complete when it captures all aspects of construct. 
Determining the responsiveness of the metrics is beyond what can be accomplished in 
this study. Capturing the responsiveness dimension will require a more in depth 
understanding of the HCA organization structure as well the intricacies and 
responsibilities of the different positions within that organization. To reiterate from the 
literature review, a responsive measure reflects actions that a manager can directly 
influence. Most of these metrics encompass actions resulting from the work of an entire 
enterprise and therefore it is difficult to ascertain at what level they are responsive or not 
responsive. 

1. Customer 


Metric 


Customer Satisfaction Index 


Can Customer Satisfaction be measured effectively? It depends on several things. 
For instance: Who is filling out the survey? Is it the same person as last year? How does 
this person typically fill out a survey: accurately, harshly, or lenient? Is this the same 
survey that was given out last year? If not, how similar is it to last years, does it cover 
the same general customer satisfaction themes? Does a survey even go out every year or 
was last year’s survey responses used for this year’s metric? What was your response 
rate? All of these questions offer a look into whether or not the Customer Satisfaction 
Index metric can be an effective measure as well as a tool to analyze trends from the 
metric. 


All of the aforementioned questions about the surveys identify possible risks in 
using the surveys in trend analysis. Is a survey better than none? Absolutely. Any one 
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of the factors identified in the questions by itself is not likely to affect drastically the 
validity of the Customer Satisfaction Index, but put together, they can throw off the 
baseline and trend analysis. 

The questions mainly address the issue of objectiveness. Surveys are inherently 
subjective, but if they are performed in good faith, address the four customer satisfaction 
themes, are reported on accurately, and have a high response rates, then they should be an 
effective measure of customer satisfaction. 

Is the Customer Satisfaction Index complete; does it capture all the relevant 
attributes of achievement? If the survey covers fully the four customer satisfaction 
themes required, as well as goes to and is received back from a large number of 
customers (high response rate), as described above, then the survey should capture all the 
attributes of Customer Satisfaction. The interviewees stated that customer response rate 
was not a problem. However, there is currently no required response rate delineated in 
the PMG setting the bar for acceptable survey results. 

2. People 


Metrics 

Employee Survey 


Workforce Stability 


Continuous Learning 


DAWIA Certification 


Acquisition Professional Community 


All of the questions that can be asked about the Customer Satisfaction Survey can 
be asked about the Employee Survey and they lead in the same direction. The 
characteristics of an objective and complete employee survey can be gleaned from the 
above discussion of the customer satisfaction survey. That is, surveys are inherently 
subjective, but that characteristic can be minimized through the appropriate methods; for 
instance, requiring a high response rate and accurate reporting. No specific percentage 
for response rate was discussed in the interviews, but one interviewee stated, “Need to 
put in place a certain percentage that has to respond for it to be valid. The survey is 
anonymous, so it is hard to control.” Also, the survey is complete as long as it measures 
all necessary attributes of employee satisfaction. One interviewee mentioned that their 
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employee survey is sent to the entire SYSCOM, not just the HCA, so it is not just 
measuring the HCA’s employee satisfaction. 

Workforce Stability is an objective measure unless estimates are used; however, 
this metric is not complete as it currently stands. The metric does not capture significant 
elements of workforce stability; rather it captures just a limited scope. It should be 
possible to effectively measure Workforce Stability, but that is not what is being 
measured by using this metric. This metric measures the percent difference between the 
authorized end-strength and the actual number of employees onboard as of the end date 
of the quarter. This measures how “full” the organization is, not how stable the 
workforce is. Even with several data points of this metric creating a trend, it still is not 
very useful unless you are assessing how full an organization is over time. The measure 
tangentially relates to stability. If during a period of time the percent difference stayed 
relatively stable, it could be that there was stability in the workforce. It could also mean 
that each time you measured stability using this formula, the number of employees and 
end-strength remained about the same, but that there was a 100 percent turnover of the 
workforce. This is not a stable workforce, it has high turnover which is almost certainly 
detrimental to the organization’s effectiveness. 

The next three metrics under the People Focus Area, Qualifications: (1) 
Continuous Learning, (2) DAWIA certification, and (3) Acquisition Professional 
Community can be measured effectively. The data for these metrics are pulled by DASN 
ACQ from the Data Acquisition and Control Manager Mission Information System 
database (DACM MIS database). As long as the infonnation is correctly inputted in the 
database the resulting metric should be accurate and therefore make the metric an 
objective one. These metrics also present a full picture of the level of qualifications that 
the procurement workforce making the measures relatively complete. 
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3. Process 


Metrics 

E2E Metric 


P-Card Delinquency 


DD1716 


Interest Penalties 


Cycle Time 


Consolidate Service Contract 


The E2E metric can be effectively measured. This metric is objective because 
information for the metric is taken from the same database for each SYSCOM and 
therefore, can be independently verified. As is the case with most of the metrics that are 
being collected from databases, the accuracy of the infonnation entered into the database 
is paramount to getting an accurate reflection of the SYSCOM. The P-Card Delinquency 
metric is affected by the same circumstances as the E2E metric. It is effectively 
measured through the Navy’s e-Business Office and is objective. The Consolidate 
Sendee Contract metric is collected by DASN ACQ from the Procurement Management 
Reporting System (PMRS). All three metrics; E2E , P-Card Delinquency, and 
Consolidate Service Contract, are narrowly focused and inclusive, making the 
measurement of the metric complete. 

The Efficiency Improvement metric of DD1716 is collected by DASN ACQ from 
the Navy-Air Force Interface (NAFI) website. This metric is relatively objective because 
the data is pulled from the NAFI website, but there is a tinge of subjectivity surrounding 
what the author views as the potential for gaming the system, which is discussed in a later 
section. This metric, if the receiving date entered into the NAFI website is agreed upon, 
is complete in its measure of the efficiency of DD1716 resolution. 

The issues of objectivity and completeness for the Cycle Time metric are similar 
to the DD1716 metric. This metric is reported by the HCAs instead of being collected 
from a database, increasing its subjectivity. Since there are several determinates when 
deciding the actual start and end date of the metric, the chances of gaming increase. It is 
a relatively complete measure of the start to the end of acquisition time. 
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4. Financial 


Metrics 

Procurement Direct/Indirect Ratio 


Industry Spend Analysis—Competition 


Industry Spend Analysis—Small Business 


Industry Spend Analysis—Commercial Items—Actions 


Industry Spend Analysis—Commercial Items—Dollars 


Industry Spend Analysis—Perfonnance Based Services—Actions 


Industry Spend Analysis—Perfonnance Based Services—Dollars 


The Procurement Direct/Indirect Ratio metric is a measure that has potential. As 
stated above in the “Aligning with Strategy” section, this metric is designed to measure 
how much support is needed to carry out the procurement function for each SYSCOM. 
The metric is likely to differ between the SYSCOMs. That is appropriate. One 
SYSCOM could mainly be making small purchases and the other purchasing Aircraft 
Carriers. The two SYSCOMs in this example would have different ratios given a need 
for different levels of various activities and processes. Even if the ratios differ across 
SYSCOMs, a trend analysis within the SYSCOMs is beneficial. 

Given how this metric is collected, it is not totally objective. There needs to be 
very specific instruction on which positions should be reflected in this metric. Precision 
and accuracy in collecting the data for this metric will ensure it is objective. One 
interviewee stated, “Direct/Indirect [is] a little bit subjective. The infonnation is not easy 
to get.” Another interviewee stated, “Does anyone track procurement salary? 

Information is there, but not broken down into sections like procurement and further into 
direct and nondirect.” Reflecting the need for clarification on definitions in regards to 
this metric, one interviewee stated, “There is confusing language in the definition of 
tenns, people need to clearly understand what is being asked for.” This metric gives a 
relatively complete picture of how much indirect support is needed by SYSCOM in order 
to perfonn their level of procurement. 

The Industry Spend Analysis—Competition metric can be effectively measured. 
All of the information needed to calculate the metric is in the Procurement Management 
Reporting System (PMRS) and is pulled by DASN ACQ which makes it an objective 
measure. The same also holds true for the following five Industry Spend Analysis 
metrics: (1) Small Business, (2) Commercial Items — Actions, (3) Commercial Items — 


31 






Dollars, (4) Performance Based Sendees Acquisition Contracting—Actions and (5) 
Performance Based Services Acquisition Contracting — Dollars. These metrics are all 
collected from PMRS by DASN ACQ. All of these metrics are complete; they capture all 
the relevant information that is needed to get a useful view of how the Navy’s 
procurement function is performing in each of these areas. 


5. Value 


Metrics 

Performance Unit Costing—Large Contracts 


Performance Unit Costing—Simplified Acquisition Procedures 


The Performance Unit Costing (PUC) metrics, Large Contracts and Simplified 
Acquisition Procedures, can be measured, but the degree of the measure’s precision is in 
question. First, the Customer Satisfaction Index is used in the formula. This is not a 
completely objective variable, so it adds some subjectivity into these metrics. This index 
is squared in the formula to increase its affect on the output of the equation. 

Second, because the data available for procurement salary dollars to the 
SYSCOMs differs, the use of reasonable estimation is an option given by the PMG. This 
makes the likelihood of independent verification questionable. One interviewee stated, 
“Procurement salaries can be subjective, [you] need an unbiased source that they can pull 
from.” 

The contract action dollars, variable “O,” is defined as “The total net value of 
obligations under contract actions” (PMG, vl4.3, p. 16). This is a relatively objective 
measure with no estimation needed. 

The PUC metrics are intended to be a complete measure of value and specifically 
balance quality, cost and productivity. The metrics, although somewhat abstract and not 
very meaningful without trend analysis, do blend all three of these elements together 
using the fonnula. The HCA representatives generally understood the concept and why 
the formula was created the way it was, but were concerned with the meaning of the 
output. 
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C. ARE THE METRICS LINKED TO VALUE? 

All of the HCAs interviewed thought that feedback from DASN ACQ was 
significantly lacking on (1) the DoN Procurement Metrics in general and (2) specific 
feedback on the metrics DASN ACQ collects on each of the SYSCOMs. In the HCAs 
view, if they are not collecting the data for a metric, then it does not even exist because 
they have not seen any feedback on the fourteen metrics that DASN ACQ collects. 
Therefore those fourteen metrics are of little value to the HCAs, regardless if they have 
value for DASN ACQ. 

Being the POCs for the DoN Procurement Metrics and collecting the metrics is a 
collateral duty for all of the POCs. They receive no additional resources to aid in the data 
calls. The collection of these metrics is a time-consuming process. Collecting the data 
for the metrics expends valuable resources and it is not clear to the HCAs if the “cost” is 
worth the benefit or value the metrics provide to the SYSCOMs and DASN ACQ. 

An issue brought up in one of the interviews was whether or not DASN ACQ had 
any idea of how much it cost to develop, monitor and collect the data on the metrics. So, 
what is the benefit [value] of the metrics and is this value gained from the metrics worth 
the resources expended in managing, monitoring, and collecting them? The answer to 
this question was not completely discernible from any of the interviews conducted with 
the HCA representatives because for the most part they were asking of what value are the 
metrics to DASN ACQ. 

The majority of the metrics are collected on an annual basis. Of the seven metrics 
that the HCAs report on, only one is reported quarterly, Workforce Stability. Reporting 
on the specific metrics annually makes sense because of the nature of procurements due 
to yearly appropriations from Congress and the fiscal year cycle. However, there are two 
mind-sets at odds with each other. The first is that sometimes employees are not in their 
positions long enough for annual metrics to be of any value in a managerial sense 
because managers might only see one or two sets of annual data. Seeing limited data 
does not provide managers with much value and does not give them the ability to make 
decisions and manage based on the variances observed. The counter to this argument is 
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that if there was past data, then that could be looked at in conjunction with current data. 
The second mind-set is, as stated above, collecting the data is not a minor task and 
answering data calls more than once a year for specific metrics would be extremely time- 
consuming if not impossible without additional resources in place. 

The PUC metrics {Large Contracts & Simplified Acquisition Procedures ) were 
discussed more frequently during the interview process than other metrics. The general 
concept of the metric was understood, but the HCA representatives did not see much 
value in the output. An interviewee stated, “Concept of PUC is nice, but real world 
application is questionable. Goes back to how it is used. [A result of] .003 down, why 
did that happen?” The HCAs need more feedback on the two PUC metrics and what the 
results mean. The use of the Customer Satisfaction Index was a point of contention with 
one SYSCOM. There were two parts to this point. One, this metric is already used 
independently of the PUC metrics. Second, the weighting of the Customer Satisfaction 
Index is increased by squaring it which increases the “role” of this subjective and 
incomplete metric. 

Another issue regarding these two metrics {PUC-Large Contracts & PUC- 
Simplified Acquisition Procedures) involved the use of contract action dollars in the 
formula. Two of the HCA representatives from the SYSCOMs would prefer to use 
“contract actions” instead of “contract action dollars” as one of the variables in the 
metric. One other SYSCOM was strongly against “contract actions” replacing “contract 
action dollars.” This SYSCOM felt this change would not create a more accurate 
reflection of their performance in this area, stating, “Clearly do not use actions.” Another 
SYSCOM felt that there was not a clear answer, stating, “When you use dollars you 
could be seen as more efficient without being. [There is] not much control over number 
of actions because a lot of control is set by Congress’ appropriations.” 

Indication of the metric’s link to value was the SYSCOMs view that the Navy 
Procurement metrics are not the specific ones that each individual SYSCOM would have 
come up with to tell the story of the health of their specific organization, but that was not 
the purpose of these metrics. One SYSCOM stated, “We wouldn’t have come up with all 

of these metrics if you asked, ‘how [our SYSCOM] is doing?”’ 
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D. ADDITIONAL ANALYSIS 


It became clear to the author through the evaluation of the metrics that there were 
two different types of analysis, design and use. The evaluation of the metrics’ alignment 
with strategy, effective measurement (objectivity and completeness), and link to value, all 
fall under the “design” of this management control system. The analysis of design 
captures the fundamental structure of the system and answers the questions: Is it valid? Is 
the design sufficient enough for the system to work? This section discusses findings 
related to the “use” of the metrics. This includes, but is not limited to, issues with 
communication and infonnation gathering. 

1. Customer 


Metric 


Customer Satisfaction Index 


Since the individual SYSCOMs are responsible for the customer survey that 
eventually is used as input into the Customer Satisfaction Index, it is important that they 
understand all of the factors that affect the survey’s validity. Not all of these factors can 
be controlled. There are two issues which are more relevant to as well as easier 
controlled by the SYSCOMs. The issues are: 1) ensuring that the survey goes out to the 
customers every year and 2) ensuring that the DASN ACQ general customer satisfaction 
themes are covered. These two issues, out of all the ones mentioned, are the most 
important in receiving relevant data for the metric. 

2. People 


Metrics 

Employee Survey 


Workforce Stability 


Continuous Learning 


DAWIA Certification 


Acquisition Professional Community 


What was striking about the Employee Survey is what was learned through the 


interviews with the SYSCOM points of contact for the DoN Procurement Metrics. The 
response rate for the Employee Survey is less than the targeted. One SYSCOM gave an 
estimated response rate of about 9 percent and another of about 2.5 percent. It is striking 
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to find that out of four-hundred surveys sent out, ten were completed. It does not matter 
how well the survey is written, employee satisfaction cannot be effectively measured 
with such low response rates. 

An issue with the Workforce Stability metric, as well as with others, is the 
availability of the infonnation. It would seem simple enough to find out how many 
employees are onboard and how many are authorized, but this is not the case for every 
SYSCOM. A few of the SYSCOMs indicated that this seemingly undemanding task is 
extremely time consuming for them. This goes back to the issue of expending effort and 
resources to what cost and what benefit. This also affects the objectiveness of this 
metric. Is the data accurate? One interviewee indicated this about the military portion of 
the Workforce Stability metric: “[We] just do not have the military data, we could 
estimate it, but then it becomes subjective.” 


3. Process 


Metrics 

E2E Metric 


P-Card Delinquency 


DD1716 


Interest Penalties 


Cycle Time 


Consolidate Service Contract 


When measuring the number of days turnaround time ( DD1716 metric) from 
receipt to disposition there may be some room for gaming the system with respect to 
when exactly the Contract Deficiency Report was received and acted upon. The different 
players in the process may have different views on when that occurred and the 
individuals who control the input into the NAFI website will win that argument and could 
possibly be inputting data that is not 100 percent accurate. 

The same sort of “gaming” problem exists with the Cycle Time metric. It again, is 
a “days” measure and it relies on four definitions of when the days should start to be 
counted and two definitions of when the days should stop being counted. The Cycle Time 
start date is defined as when the earliest of the four events listed in the Metrics Guide 
occurs (date of Procurement Planning Conference (PPC), date of planning meeting 
equivalent to a PPC, date specialist begins work on procurement, or Procurement Request 
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(PR)/Procurement Initiation Document (PID) receipt date) and the end date is listed as 
either the contract award date or for work in process, the current date if more than 365 
days have elapsed since the start date. This may seem relatively straightforward, but 
there appear to be ways to manipulate the Cycle Time metric. For instance, the actual 
start date can be manipulated by the individual recording it to start later in order to appear 
to have a shorter cycle time. 


4. Financial 


Metrics 

Procurement Direct/Indirect Ratio 


Industry Spend Analysis—Competition 


Industry Spend Analysis—Small Business 


Industry Spend Analysis—Commercial Items—Actions 


Industry Spend Analysis—Commercial Items—Dollars 


Industry Spend Analysis—Perfonnance Based Services—Actions 


Industry Spend Analysis—Perfonnance Based Services—Dollars 


What is difficult with regards to the Procurement Direct/Indirect Ratio metric, as 
well as others, is that the Information Technology and Human Resources system that is in 
place was not designed to generate these metrics. In one of the interviews it was 
indicated that the metrics working group which created these metrics worked with IT and 
HR contacts to come up with “friendly” metrics that would be easier to produce. 
However, several of the interviewees indicated that getting the necessary data was 
difficult at best and that the information is not broken down into the categories that are 
required by the metrics. For example, the salary data required for the Procurement 
Direct/Indirect Ratio is not available already categorized as direct or as procurement 
dollars. Time is limited and the interviewees indicated the extremely cumbersome nature 
of producing the required metrics. The difficulty in collecting this metric may vary 
among the SYSCOMs depending on what resources they have available and how their 
data is kept. 

The ability to obtain the HCA total procurement salary dollars, for the 
Procurement Direct/Indirect Ratio metric, was described as a complex process. In the 
glossary section of the PMG, it states that “data available to activities vary greatly, no 
specific way of calculating procurement salary dollars is mandated. These approaches 
should provide a reasonable estimate of actual labor costs, but may be based on models 
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and/or average rates, etc” (PMG, vl4.3, p. 20). This means that each HCA could 
calculate procurement salary dollars in a different manner. If the HCAs are internally 
consistent in the manner they collect the salary data and their method of estimation then 
trends can be analyzed. They can be analyzed because the consistency of the collection 
method makes the data points comparable. This will help take the subjectivity out of the 
total procurement salary dollars variable. 

Table 2 is a summary of the findings regarding the three questions described as 
the general framework for evaluating the metrics. There are varying levels of objectivity 
and completeness as previously discussed for each individual metric. For purposes of 
providing a summary table, “yes” or “no” was used to describe whether or not the metric 
was deemed objective or complete. The issue of the metric’s link to value was covered 
only for the seven metrics the HCAs collect. As stated above the interviewees are of the 
opinion that the fourteen metrics collected by DASN ACQ are currently of little value to 
the HCAs. The remaining seven metrics provide some value, but it is difficult to glean 
the extent of this value because none of the interviewees indicated specifically how and 
when they are able to use them. 
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FOCUS 

AREA 

TIER 1 METRIC 

TIER 2 METRIC 

TIER 3 
METRIC 

TIER 4 
METRIC 

LINKED TO 
STRATEGY 

OBJECTIVE 

COMPLETE 

RESPONSIVE 

LINKED TO 
VALUE 

Customer 

Customer Satisfaction Index 




YES 

NO 

NO 


YES 

People 

Human Capital Index 

Employee Survey 



YES 

NO 

NO 

NO 

Workforce Stability 



YES 

YES 

NO 

Emma 

Qualifications 

Continous 

Learning 


YES 

YES 

YES 

DAWIA 


YES 

YES 

YES 



Acquisition 

Professional 

Community 


YES 

YES 

YES 



Process 

Process Improvement Index 

E2E Metrics 



YES 

YES 

YES 



Efficiency 

Improvement 

P-Card 

Delinquency 


YES 

YES 

YES 





YES 

NO 

YES 



Interest 

Penalties 


YES 

YES 

YES 



Cycle Time 



YES 

NO 

YES 


SOMEWHAT 

Consolidate Services 
Contract 



YES 

YES 

YES 



Financial 

Contribution Index 

Procurement 



YES 

NO 

YES 


SOMEWHAT 

Industry Spend 
Analysis 

Competition 


YES 

YES 

YES 


Small Business 


YES 

YES 

YES 



Commercial 

Items 


YES 

YES 

YES 



Dollars 

YES 

YES 

YES 



Performance 
Based Services 
Contracting 

Actions 

YES 

YES 

YES 



Dollars 

YES 

YES 

YES 



Value 

Product Unit Costing (PUC) 

Large Contracts 



YES 

NO 

YES 

Simplified Acquisition 
Procedures 



YES 

NO 

YES 


NO 


Table 2. DoN Procurement Metrics Evaluation Summary 

Overall, the author’s conclusion is that the metrics are not providing the type of 
value that performance metrics should for managers in an organization. Based on the 
interviews, it does not appear that the metrics are being used to make any decisions about 
the organizations and how they operate. This is due to the fact that fourteen of the 
twenty-one metrics are not seen, so they are in essence off the table, and the other seven 
have varying levels of value to the HCAs, if any at all. The seven metrics that the HCAs 
report on (Table 3) are of limited value for two primary reasons. First, it was indicated in 
the interviews that estimations of variables may be used in order to report the metric. This 
not only creates subjective measurements but also causes inconsistency over time 
decreasing the validity of analyzing trends. Second, while annual metrics are a plus 
because they lessen the amount of resources the HCAs have to expend, they are not of 
much value when decision makers rotate to a new position and maybe only see one or 
two sets of completed metrics. 
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Focus Area 

Metric 

Customer 

Customer Satisfaction Index 

People 

Employee Survey 


Workforce Stability 

Process 

Cycle Time 

Financial 

Procurement Direct/Indirect Ratio 

Value 

PUC—Large Contracts 


PUC—Simplified Acquisition Procedures 


Table 3. HCA Collected Metrics 


5. Analysis not Specific to One Focus Area 

Since the HCAs collect and report on seven of the DoN Procurement Metrics, this 
focuses the SYSCOMs on seven of the twenty-one metrics. This being said, it is 
important to note that not having to collect the data for the other fourteen metrics is a 
very good thing in the HCAs view. They would just like feedback on them. “All the 
time [we have] given information, we have never received feedback,” was a comment 
during one of the interviews. When another HCA representative was asked if the metrics 
help them in any way, the response was, “They could, but [we receive] no feedback.” 
Some even thought it valuable to see the outputs of the other SYSCOMs, realizing that it 
is difficult to compare due to the differing nature of the procurements. Nonetheless, a 
few of the interviewees were still curious. One interviewee expressed the concern that, 
“comparing is dangerous.” 

Feedback was mentioned as part of the Cybernetic Feedback Model discussed in 
Chapter II. The feedback specific to the individual metrics deals with the variance from 
the standard created or the original baseline. Besides the Cybernetic Feedback Model 
type of feedback, which offers insight into deviations from the standard, the HCAs desire, 
as noted above, more general feedback on the status of the metrics program. The 
interviewees asked if the procurement metrics are going to be cheer leaded (have a 
champion) and supported by DASN ACQ. If DASN ACQ does support the metrics then 
the interviewees are also asking what resources will be available to the HCAs in order to 
better comply with and complete the data calls in a timely and accurate manner? One 
quote from the interviews addresses this point: “If you are going to do it [use the 
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metrics], do it consistently and bring it to attention, that will bring more meaning to it.” 
The idea of consistency is reflected in another interview, “stay consistent ...there seems 
to never be consistency.” 

Overall the feeling was that the metrics, while not the ones each specific 
SYSCOM would have picked to explain their individual success, could have some value 
if they got the appropriate feedback from DASN ACQ. One of the interviewees 
expressed the idea that DASN ACQ should provide feedback in the following manner, 
“We’re seeing a downward trend [in a specific area/metric], and you may want to look at 
this, this, and this.” 

Right now there is not the sense that DASN ACQ is there to help the SYSCOMs. 
The SYSCOMs need to see a culture of “we’re trying to help you” from DASN ACQ to 
the SYSCOMs. Along the same lines one interviewee stated, “[We] need it as a system 
to help the SYSCOMs, not the SYSCOMs fearing they would get in trouble. Show how 
they are benefiting the SYSCOMs.” 

The HCA representatives stated that the fourteen metrics collected by DASN 
ACQ are of little or no use to the HCAs because they aren’t getting any feedback on 
them. DASN ACQ has indicated that part of this reason is because the set of metrics is 
not a finished product and DASN ACQ feels reluctant to put out all of the results of their 
data pulls when they are not fully convinced that the metrics are the correct ones. 
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V. CONCLUSION 


A. SUMMARY 

The Department of the Navy Procurement Metrics show a clear link to the 
intended strategy of the five Focus Areas. Each metric li nk s back to their respective 
focus area and is aligned with Department of the Navy as well as Department of Defense 
strategies and goals. As stated in the literature review, when a metric is created, the 
managers creating it must know that by doing so they are telling their employees what is 
important. The simple fact that something is being measured speaks volumes as to its 
importance. 

The motivation behind each one of the metrics is important. Each metric is a 
piece in the puzzle that when put together should have the ability to provide an accurate 
picture of the health of the Navy’s procurement function. The metrics dashboard has the 
potential to be a great management by exception tool. However, this tool is only 
effective when those using it have confidence that the metrics in place are the correct 
ones linking to each Focus Area, have been collected and reported on accurately, are 
responsive to manager actions and decisions, provide the necessary information for 
managers to make those decisions, and have appropriate targets set for each metric. The 
proper use of the metrics dashboard by top management is not to analyze every area, but 
to just make note of areas that are not “green” and ensure that people execute a plan of 
corrective action if they have control to do so. 

Each of the metrics has had a preliminary analysis done on it that has indicated 
that all of them align with the intended strategy. The analysis continued with an 
evaluation of their objectivity and completeness. The level of objectivity and 
completeness varied. More analysis has to be done from a procurement expert’s 
viewpoint to see what can be done to minimize the subjectivity that was found to exist in 
some of the metrics, specifically the customer and employee surveys, DD1716, Cycle 
Time, and the two PUC metrics. Most of the metrics were found to be complete; they 
fully explained what it is they set out to measure, but some are not (i.e., Customer 
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Satisfaction Index and Employee Survey). One of those which is not complete is 
Workforce Stability. It is clear that this metric needs some rework as mentioned in 
section B of Chapter IV. The two survey metrics were evaluated as incomplete mainly 
because of the subjectivity issues listed in Chapter IV and also because of the response 
rate. An employee survey that has a 2.5 percent response rate is highly unlikely to give a 
complete picture of employee satisfaction. As stated before, responsiveness was not 
evaluated by the author. 

The value of the metrics to procurement managers was also analyzed. Most of the 
conclusions drawn about the value of the metrics came from the interviews conducted 
with the HCA representatives from six of the ten major SYSCOMs. It is unclear as to 
whether other personnel within the HCAs use the current metrics in a way that has value 
to them or have ideas for certain metrics they would like to use. Currently, according to 
the interviewees, fourteen of the twenty-one metrics provide little value to the HCAs 
because in their view there is no feedback on the metrics that are produced by DASN 
ACQ. The remaining seven metrics do not provide infonnation that is of great value to 
the HCAs either. This is likely to remain the case until a trend can be analyzed. 

The idea of additional metrics is not one that the HCA representatives embraced. 
This goes back to the issues of value provided and cost to provide that value. One quote 
from an interview relays this point fairly succinctly, “Any additional metrics need to be 
evaluated on how much work that it is going to require and of what value it is. Even 
collection of how many people you have compared to end strength [is cumbersome].” 

For a few of the HCA representatives, the opposite idea, getting rid of the metrics, was 
something they wouldn’t lose any sleep over. 

B. RECOMMENDATIONS 

If the DoN Procurement Metrics are to be used and be successful, they need a 
champion. Partial buy in will only prolong the period of indecision on the metrics, 
engender confusion as to the status and the purpose of the metrics, and potentially waste 
valuable time and resources that could otherwise be used more effectively. 
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The Procurement Metrics themselves are generally good measures of the Navy’s 
Procurement Function. The purpose of these metrics is to roll up data from all of the 
major SYSCOMs into a usable diagnostic tool. There is no perfect set of metrics that 
every SYSCOM would have agreed on as telling a complete and accurate picture of their 
procurement activities. A recommendation in this area would be to contact the 
appropriate person at each HCA to find out which of the metrics they (the HCA) deem 
valuable for her or his SYSCOM. It could be as simple as a table to be completed like 
the one below. The table does not list all metrics or SYSCOMs. It just provides an 
example of format. The actual table would list all twenty-one metrics and each of the ten 
SYSCOMs, who would mark the valuable metrics: 



SYSCOM 1 

SYSCOM 2 

SYSCOM 3 

SYSCOM 4 

SYSCOM 5 

P-card Delinquency 

X 

X 

X 

X 


E2E 


X 


X 


Cycle Time 



X 


X 

Table 4. 

Sample SYSCOM Assessment of] 

Metric Value 


Feedback has to be initiated by DASN ACQ with respect to the metrics. This 
paper may offer DASN ACQ some insight into the metrics themselves, the issues that the 
HCAs have with specific metrics, as well as the metrics initiative in general. Right now 
the feedback loop is not connected, it is open. The HCAs need a sense of the status of the 
metrics and feedback on not only the fourteen metrics that are being collected by DASN 
ACQ, but all the metrics. 

If DASN ACQ decides to continue with the DoN Procurement Metrics initiative, 
then the HCAs desire the necessary resources to do an effective job at collecting and 
reporting the metrics. Again, the level of effort currently expended and resources already 
available to the HCAs most likely differ from SYSCOM to SYSCOM. Two interviewees 
from separate SYSCOMs specifically mentioned that larger SYSCOMs have contractors 
to help with surveys while they did not. 
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A new working group needs to be established. This working group should not 
only be comprised of the HCA representatives for the metrics, but also some of the 
managers in the HCAs who would specifically be using and benefiting from the metrics. 

A metric by metric evaluation can be done in much the same way as this paper evaluated 
the metrics. This evaluation would be done by operators and managers who have been in 
the procurement field and who know the ins and outs of the processes. This means that 
they could take what has been started here and fine tune it because of their experience 
and expertise in the procurement field. 

Several issues should be addressed when this working group meets: 

1. Reduce the subjectivity of the indicated metrics (Customer Satisfaction Index 
and Employee Survey ) to an acceptable point. 

2. Address the completeness of the metrics with specific emphasis on Workforce 
Stability. 

3. Ascertain the actual metric responsiveness to manager and employee 
decisions. 

4. Discuss the nature of the feedback that will be supplied to the HCAs. 

5. Determine what additional resources are available to the HCAs from DASN 
ACQ in order to provide the most accurate and useful infonnation. 

6. Discuss the definitions of the variables used in the metrics, specifically those 
collected by the HCAs, in order to reduce any guesswork. 

7. Discuss how the actual data needed is stored and what has to be done to 
categorize and retrieve that data to put it in a usable form for input into the metric 
formulas. 

Addressing these points by DASN ACQ and the HCAs is likely to provide very 
meaningful information on the next step for the metrics initiative. 

The results of this paper and especially the outcome of this new working group 

should be pivotal in deciding the future of the DoN Procurement Metrics. An improved 

understanding of the metrics as well as an improved set of metrics can only be achieved if 
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this issue is approached openly with candid discussion of the issues with no fear of 
repercussions. DASN ACQ needs to reassure the HCAs that they are there to help. Buy 
in at the top should have a positive trickle down effect. That coupled with a renewed 
sense of purpose and understanding of the metrics has the potential to transition these 
metrics into the Diagnostic Control System it intended to be. 
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